A Back-Propagation Algorithm with Optimal Use of Hidden Units

This paper presents a variation of the back-propagation algorithm that makes optimal use of a network hidden units by decr~asing an "energy" term written as a function of the squared activations of these hidden units. The algorithm can automatically find optimal or nearly optimal architectures necessary to solve known Boolean functions, facilitate the interpretation of the activation of the remaining hidden units and automatically estimate the complexity of architectures appropriate for phonetic labeling problems. The general principle of the algorithm can also be adapted to different tasks: for example, it can be used to eliminate the [0, 0] local minimum of the [-1. +1] logistic activation function while preserving a much faster convergence and forcing binary activations over the set of hidden units. PRINCIPLE This paper describes an algorithm which makes optimal use of the hidden units in a network using the standard back-propagation algorithm (Rumelhart. Hinton & Williams, 1986). Optimality is defined as the minimization of a function of the "energy" spent by the hidden units throughtout the network, independently of the chosen architecture, and where the energy is written as a function of the squared activations of the hidden units. The standard back-propagation algorithm is a gradient descent algorithm on the following cost function: P 0 C = I I (dijOij)2 [1] j where d is the desired output of an output unit, 0 the actual output, and where the sum is taken over the set of output units 0 for the set of training patterns P. 519